ToF/RGB Sensor Fusion for 3-D Endoscopy

Acquisition of 3-D anatomical structure in minimally invasive surgery is an important step towards intra-operative guidance. In this context, the rst prototype of a Time-of-Flight/RGB endoscope was engineered for simultaneous range and color data acquisition. Intrinsic and stereo camera calibration are essential to achieve an intuitive visualization of colored surfaces. Due to the early prototype stage, inhomogeneous illumination and low resolution (64×50 px) complicate the calibration signi cantly. To overcome these challenges, we propose a fully automatic multiscale calibration framework using a self-encoded marker for checkerboard detection. A rst application demonstrates the feasibility of intra-operative measurement. Using our calibration scheme, we achieved a reprojection error of less than 0.7 px for the Time-of-Flight camera and 0.5 px of the RGB camera. Our framework eases calibration and enables future applications to use combined range and colored data.